1. Field of the Invention
The invention relates to the field of digital image processing and more specifically to blemish control in solid-state CCD/CMOS sensors that are subject to pixel defects.
2. Description Relative to the Prior Art
Pixel defects in solid-state sensors such as CCD/CMOS sensors have been a common problem. The output from a CCD/CMOS sensor is subject to artifacts due to bad pixels. The visual effect of bad pixels can be very annoying as bad pixels manifest as black, white or gray dots in captured images. With sensor sizes increasing into the mega-pixel range, pixel defects are almost inevitable. Because it is very hard to produce defect-free sensors, it has been common practice for CCD/CMOS sensor manufacturers to market sensors with a small amount of defective pixels. These slightly defective sensors are often shipped with a record listing the locations of defective pixels.
Sensors with a slight defect might be tolerable for some applications if care is taken to mitigate the effect of impairment. A common solution to mitigate the effect of pixel defects involves the registration of the bad pixel locations before the product is shipped and involves an algorithm to produce the proper pixel value for the corresponding bad pixels of the captured image. For example, Kodak, a sensor manufacturer, is marketing its sensors with various defect classes ranging from defect free to some point defects, cluster defects and column defects. Because the locations of bad pixels are random, a record of these locations must be kept with each individual device. Usually, the record is transferred to non-volatile memory in the sensor device. If the record is lost, however, the sensor device must be re-tested to obtain the locations of the bad pixels. Once the locations of the bad pixels are known, the corresponding pixel values should be replaced by values estimated from surrounding pixel values. Interpolation can be used to obtain the estimated values.
Pixel defects can be classified in a variety of ways. According to one model, there are three types of pixel defects: stuck high, stuck low and abnormal sensitivity. When a pixel is stuck high (white pixel), its output is always very high regardless the input light intensity. When a pixel is stuck low (dark pixel), its output is always very low regardless of the input light intensity. The pixel subject to abnormal sensitivity produces an output level different from (higher or lower than) the normal pixels by a certain amount. To further illustrate, a pixel with the stuck high defect manifests itself as a very noticeable artifact (bright spot in a gray or dark area), which is most visually objectionable in a gray or dark area, particularly in a monochrome sensor. A pixel with stuck-low defect also produces a noticeable artifact (dark spot in a gray or bright area). A pixel with the abnormal-sensitivity defect is relatively more tolerable.
Another traditional method of dealing with the defective pixels involves a defect table with predetermined locations of the defective pixels. A description of this method can be found in U.S. Pat. No. 5,008,739, “Real-Time Digital Processor for Producing Full Resolution Color Signals from a Multi-Color Image Sensor,” by Hibbard. In that disclosure, a defect concealment circuit is incorporated to estimate the values for defective pixels, the locations of which are predetermined and placed in a defect table (column 6, line 39).
Another existing system to cope with the bad sensor pixels involves the application of a sequence of random test pictures. After each test picture is captured, each pixel of the sensor is compared to its neighboring pixels. Applying a function (see below), statistical data is collected to determine whether a pixel is defective. The probability of finding pixel defects increases as more test pictures are used. This method was developed by Y-P Tan and T. Acharya. This system is described in more detail in “A Robust Sequential Approach for the Detection of Defective Pixels in an Image Sensor,” published in Proceedings of IEEE Conference on Acoustics, Speech and Signal Processing, vol. 4, pp. 2239–2242, March 1999. A minimum neighboring pixel difference (MND) is calculated as:y(i,j)=min{|I(i,j)−I(m,n)|} for(m,n)εG(i,j)where G(i,j) denotes the locations of the pixels within the neighboring support around pixel (i,j). I(i,j) is the output of pixel (i,j), and I(m,n) is the output pixel at (m,n). They examine the conditional probability density function p(y/z), where z is the pixel value from the neighborhood of (i,j) that results in the MND for the underlying pixel y. The conditional probability density function exhibits very distinctive characteristics between a normal pixel and a defective pixel. When enough statistical data is collected for each pixel location, a fairly accurate decision can be made as to whether an underlying pixel has a defect, and the type of defect if it is defective. Based on simulation, they concluded that accurate defect detection could be made with a training process where sequence of 8 random test pictures is applied to the function described above. Though the method can result in accurate defect detection, it relies on the training process to form the conditional probability density function. This process may not be desirable.
Another method to alleviate the effect of defective pixels involves applying signal processing to every pixel. Proper pixel values are extrapolated from the values of neighboring pixels. The method is applied globally to all pixels whether defective or not. This method was developed by B. Dierickx and G. Meynants in “Missing Pixel Correction Algorithm for Image Sensor,” published in Proceedings of SPIE, vol. 3410, pp. 200–203, May 1998. The assumption is that an image projected through a lens or any other optical system is never perfectly sharp. Even with ideal lenses, a star image, for example, would not be projected on a single pixel. The point-like source of the star would be smeared out over a central pixel and a few neighbors. To correct the possible defective pixel, they examine 4 pixels surrounding an underlying pixel (one-dimensional processing). They form an extrapolated value for the underlying pixel from the 2 pixels on the left and the 2 pixels on the right.
The upper bound, Cmax, for the underlying pixel is defined as the maximum value among the 2 extrapolated values and the two immediate neighboring pixel values. The lower bound, Cmin, for the underlying pixel is defined as the minimum value among the 2 extrapolated values and the two immediate neighboring pixel values. The final corrected value for the underlying pixel is the median of Cmax, Cmin, and the original underlying pixel value. The signal processing is applied to every pixel whether it is defective or not. Though the method does a fairly good job alleviating the effect of defective pixels, some subtle features, however, may be altered inadvertently. This occurs because even all pixels are fixed, even good pixels.
A few US Patents granted in recent years relate to missing pixel processing for color image sensors. These patents, however, deal with an issue very different from the present invention. For example, U.S. Pat. Nos. 6,181,376 and 6,188,804 are directed to a full reconstruction of a sampled image which has missing information. In this case, the sensor does not provide full resolution samples. One such example is the Bayer pattern (RGB sampling pattern) where each line contains either R-G pixels or G-B pixels. The G pixel pattern thus represents 50% of the total pixels and each of the R and B pixels represent 25% of the total pixels. These two patents address the method of interpolating the values for the missing color pixels.
If the locations of defective pixels are known, a simple interpolation would generally do a decent job. In such a scenario, when a slightly defective sensor is incorporated into an imaging system, a simple pixel interpolation can be applied to the defective locations. This simple method can effectively reduce the artifacts caused by bad pixels. However, a defective pixel detection algorithm can result in false detections where a normal pixel is classified as a defective pixel. Moreover, the probability of false detections increases as sensor sizes increase.
A need therefore remains for an image processing method and apparatus that detects and identifies pixel defects without prior knowledge of such defects and restores the values of defective pixels in a manner that is time efficient, simple, and reliable. A need also remains for a method and apparatus that minimizes the occurrence of false detections.